Calibration of virtual reality systems

ABSTRACT

A virtual reality (VR) console receives slow calibration data from an imaging device and fast calibration data from an inertial measurement unit on a virtual reality headset. Using a model of the VR headset, the VR console identifies model locators corresponding to locators on the VR headset and generates estimated positions for locators included in slow calibration data. The VR console adjusts calibration parameters so a relative distance between estimated positions of the locators and positions of their corresponding model locators is less than a threshold value. From the estimated positions, the VR console generates calibrated positions of a reference point on the VR headset associated with images from the slow calibration data. The VR console determines predicted positions of the reference point from the calibrated positions and adjusts calibration parameters so intermediate estimated positions of the reference point are within a threshold distance of the predicted positions.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/589,755, filed Jan. 5, 2015, which claims the benefit of U.S.Application No. 62/088,085, filed Dec. 5, 2014, and U.S. ProvisionalApplication No. 62/088,088, filed Dec. 5, 2014, and U.S. ProvisionalApplication No. 61/923,895, filed on Jan. 6, 2014, all of which areincorporated by reference in their entirety.

BACKGROUND

The present disclosure generally relates to calibration systems, andmore specifically relates to calibrating virtual reality systems.

Virtual reality (VR) devices include components for determining positionand movement of a headset worn by a user. These components need to becalibrated at various times, initially due to manufacturing tolerancesand subsequently due to normal use of the system. Operating improperlycalibrated VR device may result in improper tracking of the position ormotion of the headset, which causes a dissonance between user motion andmedia presented to the user via the headset. Moreover, one or more ofthe components determining headset position and movement can losecalibration over time or with use. For example, changes in temperatureor vibration may cause a camera imaging the motion of the headset tolose calibration.

SUMMARY

Components of a virtual reality (VR) system are calibrated to maintaintracking of a VR headset associated with the VR system. The VR systemuses slow calibration data received from an imaging device and fastcalibration data received from an internal measurement unit (IMU)included in the VR headset for calibration. In some embodiments,components of the VR system may be calibrated by initially applying oneor more default parameters to the components. Based on the defaultparameters, the VR system tracks movement of the VR headset byidentifying positions associated with one or more locators included onthe VR headset. A locator is an object located in a specific position onthe VR headset relative to one or more components, such as anotherlocator, of the VR headset and relative to a reference point on the VRheadset. In some embodiments, the VR headset includes two rigid bodiesthat are non-rigidly coupled to each other, with locators included oneach of the rigid bodies for tracking the user's head position andorientation. The VR system adjusts one or more calibration parametersuntil differences between an estimated position of one or more locatorsdiffers from an observed position of the one or more locators by lessthan a threshold value.

In some embodiments, the VR system includes a VR console that receivesslow calibration data including a series of images showing a portion ofa plurality of locators on the VR headset from an imaging device. Eachimage is separated from a subsequent image in the series by an imagetime value. Additionally, the VR console receives fast calibration datacomprising one or more intermediate positions of the reference point onthe VR headset from the IMU included in the VR headset. An intermediateestimated position of the reference point is a position determined fromthe fast calibration data and may be associated with a time associatedwith an image, or a time between times associated with an image and asubsequent image from the slow calibration data. The IMU determines theintermediate estimated positions of the reference point based on datafrom one or more position sensors (e.g., accelerometers, gyroscopes)included in the VR headset. Each intermediate estimated position isseparated from a subsequent intermediate estimated position by aposition time value that is less than the image time value.

The VR console generates estimated positions for the observed locatorson the VR headset using a headset model. For example, the VR consoleuses the headset model and the information identifying positions of theobserved locators to determine a projection matrix for translating idealpositions (described by the headset model) to positions on the imageplane (described by the images of the observed locators) of the imagingdevice. The VR console uses the projection matrix to estimate positionsof the observed locators, and adjusts one or more calibration parametersto adjust one or more of the estimated positions of observed locatorsuntil relative distances between adjusted estimated positions ofobserved locators and their corresponding positions determined by theheadset model observed locations are less than a threshold value. Basedon adjusted estimated positions of the observed locators, the VR consoledetermines calibrated positions of the reference point of the VR headsetfor one or more images from the slow calibration data. The VR consoleadditionally adjusts one or more calibration parameters so theintermediate estimated positions of the reference point from the fastcalibration data are within a threshold value of predicted positions ofthe reference point determined from the calibrated position of thereference point (e.g., via curve fitting) from the slow calibrationdata. In some embodiments, components of the VR system may be calibratedsimultaneously to (1) adjust calibrations to adjust the estimatedpositions so a relative distance between the adjusted estimatedpositions of observed locators and positions of their correspondingmodel locaters is less than a threshold value; and (2) adjust theestimated positions of the reference point determined from the fastcalibration data such that a relative distance between the estimatedpositions of the reference point and positions of a model referencepoint determined from the model locators is less than the thresholdvalue.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system environment in which a virtualreality console operates, in accordance with an embodiment.

FIG. 2A is a wire diagram of a virtual reality headset, in accordancewith an embodiment.

FIG. 2B is a wire diagram of a virtual reality headset including a frontrigid body and a rear rigid body, in accordance with an embodiment.

FIG. 3 is a block diagram of a tracking module of a virtual realityconsole, in accordance with an embodiment.

FIG. 4 is a flowchart of a process for calibrating a virtual realitysystem, in accordance with an embodiment.

FIG. 5 is a flowchart of a process for re-establishing calibrationbetween two rigid bodies in a virtual reality headset included in avirtual reality system, in accordance with an embodiment.

FIG. 6 is a flowchart of a process of maintaining a positionalrelationship between two rigid bodies in a virtual reality headsetincluded in a virtual reality system, in accordance with an embodiment.

FIG. 7 is an example graph illustrating a series of calibrated positionsof a virtual reality headset, in accordance with an embodiment.

The figures depict embodiments of the present disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles, or benefits touted, of the disclosure described herein.

DETAILED DESCRIPTION

System Architecture

FIG. 1 is a block diagram of one embodiment of a virtual reality (VR)system environment 100 in which a VR console 110 operates. The systemenvironment 100 shown by FIG. 1 comprises a VR headset 105, an imagingdevice 135, and a VR input interface 140 that are each coupled to the VRconsole 110. While FIG. 1 shows an example system 100 including one VRheadset 105, one imaging device 135, and one VR input interface 140, inother embodiments any number of these components may be included in thesystem 100. For example, there may be multiple VR headsets 105 eachhaving an associated VR input interface 140 and being monitored by oneor more imaging devices 135, with each VR headset 105, VR inputinterface 140, and imaging devices 135 communicating with the VR console110. In alternative configurations, different and/or additionalcomponents may be included in the system environment 100.

The VR headset 105 is a head-mounted display that presents media to auser. Examples of media presented by the VR head set include one or moreimages, video, audio, or some combination thereof. In some embodiments,audio is presented via an external device (e.g., speakers and/orheadphones) that receives audio information from the VR headset 105, theVR console 110, or both, and presents audio data based on the audioinformation. Example embodiments of the VR headset 105 are furtherdescribed below in conjunction with FIGS. 2A and 2B.

In various embodiments, the VR headset 105 may comprise one or morerigid bodies, which may be rigidly or non-rigidly coupled to each other.A rigid coupling between rigid bodies causes the coupled rigid bodies toact as a single rigid entity. In contrast, a non-rigid coupling betweenrigid bodies allows the rigid bodies to move relative to each other. Anembodiment of the VR headset 105 that includes two rigid bodies that arenon-rigidly coupled together is further described below in conjunctionwith FIG. 2B.

The VR headset 105 includes an electronic display 115, one or morelocators 120, one or more position sensors 125, and an inertialmeasurement unit (IMU) 130. The electronic display 115 displays imagesto the user in accordance with data received from the VR console 110. Invarious embodiments, the electronic display 115 may comprise a singleelectronic display or multiple electronic displays (e.g., a display foreach eye of a user). Examples of the electronic display 115 include: aliquid crystal display (LCD), an organic light emitting diode (OLED)display, an active-matrix organic light-emitting diode display (AMOLED),some other display, or some combination thereof. Additionally, theelectronic display 115 may be associated with one or more opticalcomponents correcting one or more types of optical error (e.g., fieldcurvature, astigmatism, barrel distortion, pincushion distortion,chromatic aberration, chromatic aberration, etc.). In some embodiments,the media provided to the electronic display 115 for presentation to theuser is pre-distorted to aid in correction of one or more types ofoptical errors. Additionally, the optical components may increase afield of view of the displayed media through magnification or throughanother suitable method. For example, the field of view of the displayedmedia is such that the displayed media is presented using almost all(e.g., 110 degrees diagonal), and in some cases all, of the user's fieldof view.

The locators 120 are objects located in specific positions on the VRheadset 105 relative to one another and relative to a specific referencepoint on the VR headset 105. A locator 120 may be a light emitting diode(LED), a corner cube reflector, a reflective marker, a type of lightsource that contrasts with an environment in which the VR headset 105operates, or some combination thereof. In embodiments where the locators120 are active (i.e., an LED or other type of light emitting device),the locators 120 may emit light in the visible band (˜380 nm to 750 nm),in the infrared (IR) band (˜750 nm to 1 mm), in the ultraviolet band (10nm to 380 nm), some other portion of the electromagnetic spectrum, orsome combination thereof.

In some embodiments, the locators are located beneath an outer surfaceof the VR headset 105, which is transparent to the wavelengths of lightemitted or reflected by the locators 120 or is thin enough to notsubstantially attenuate the wavelengths of light emitted or reflected bythe locators 120. Additionally, in some embodiments, the outer surfaceor other portions of the VR headset 105 are opaque in the visible band.Thus, the locators 120 may emit light in the IR band under an outersurface that is transparent in the IR band but opaque in the visibleband.

The IMU 130 is an electronic device that generates fast calibration databased on measurement signals received from one or more of the positionsensors 125. A position sensor 125 generates one or more measurementsignals in response to motion of the VR headset 105. Examples ofposition sensors 125 include: one or more accelerometers, one or moregyroscopes, one or more magnetometers, or any other suitable type ofsensor, or some combination thereof. The position sensors 125 may belocated external to the IMU 130, internal to the IMU 130, or somecombination thereof.

Based on the one or more measurement signals from one or more positionsensors 125, the IMU 130 generates fast calibration data indicating anestimated position of the VR headset 105 relative to an initial positionof the VR headset 105. For example, the position sensors 125 includemultiple accelerometers to measure translational motion (forward/back,up/down, left/right) and multiple gyroscopes to measure rotationalmotion (e.g., pitch, yaw, roll). In some embodiments, the IMU 130rapidly samples the measurement signals and calculates the estimatedposition of the VR headset 105 from the sampled data. For example, theIMU 130 integrates the measurement signals received from theaccelerometers over time to estimate a velocity vector and integratesthe velocity vector over time to determine an estimated position of areference point (e.g., intermediate estimated position) on the VRheadset 105. Alternatively, the IMU 130 provides the sampled measurementsignals to the VR console 110, which determines the fast calibrationdata. The reference point is a point that may be used to describe theposition of the VR headset 105. While the reference point may generallybe defined as a point in space; however, in practice the reference pointis defined as a point within the VR headset 105 (e.g., a center of theIMU 130).

The IMU 130 receives one or more calibration parameters from the VRconsole 110. As further discussed below, the one or more calibrationparameters are used to maintain tracking of the VR headset 105. Based ona received calibration parameter (e.g., IMU parameters), the IMU 130 mayadjust its operation (e.g., change sample rate, etc.). In someembodiments, as further described below, certain calibration parameterscause the IMU 130 to offset an estimated position of the VR headset 105to correct positional errors that may occur when only certain portionsof the VR headset 105 are visible to the imaging device 135. In someembodiments, certain calibration parameters cause the IMU 130 to updatean initial position of the reference point so it corresponds to a nextcalibrated position of the reference point. Updating the initialposition of the reference point as the next calibrated position of thereference point helps reduce accumulated error associated with thedetermined estimated position. The accumulated error, also referred toas drift error, causes the estimated position of the reference point to“drift” away from the actual position of the reference point over time.

The imaging device 135 generates slow calibration data in accordancewith calibration parameters received from the VR console 110. Slowcalibration data includes one or more images showing observed positionsof the locators 120 that are detectable by the imaging device 135. Theimaging device 135 may include one or more cameras, one or more videocameras, any other device capable of capturing images including one ormore of the locators 120, or some combination thereof. Additionally, theimaging device 135 may include one or more filters (e.g., used toincrease signal to noise ration). The imaging device 135 is configuredto detect light emitted or reflected from locators 120 in a field ofview of the imaging device 135. In embodiments where the locators 120include passive elements (e.g., a retroreflector), the imaging device135 may include a light source that illuminates some or all of thelocators 120, which retro-reflect the light towards the light source inthe imaging device 135. Slow calibration data is communicated from theimaging device 135 to the VR console 110. The imaging device 135receives one or more calibration parameters from the VR console 110, andmay adjust one or more imaging parameters (e.g., focal length, focus,frame rate, ISO, sensor temperature, shutter speed, aperture, etc.)based on the calibration parameters.

The VR input interface 140 is a device that allows a user to send actionrequests to the VR console 110. An action request is a request toperform a particular action. For example, an action request may be tostart or end an application or to perform a particular action within theapplication. The VR input interface 140 may include one or more inputdevices. Example input devices include: a keyboard, a mouse, a gamecontroller, or any other suitable device for receiving action requestsand communicating the received action requests to the VR console 110. Anaction request received by the VR input interface 140 is communicated tothe VR console 110, which performs an action corresponding to the actionrequest. In some embodiments, the VR input interface 140 may providehaptic feedback to the user in accordance with instructions receivedfrom the VR console 110. For example, haptic feedback is provided whenan action request is received, or the VR console 110 communicatesinstructions to the VR input interface 140 causing the VR inputinterface 140 to generate haptic feedback when the VR console 110performs an action.

The VR console 110 provides media to the VR headset 105 for presentationto the user in accordance with information received from one or more of:the imaging device 135, the VR headset 105, and the VR input interface140. In the example shown in FIG. 1, the VR console 110 includes a mediastore 145, a tracking module 150, and a virtual reality (VR) engine 155.Some embodiments of the VR console 110 have different modules than thosedescribed in conjunction with FIG. 1. Similarly, the functions furtherdescribed below may be distributed among components of the VR console110 in a different manner than is described here.

The application store 145 stores one or more applications for executionby the VR console 110. An application is a group of instructions, thatwhen executed by a processor, generates media for presentation to theuser. Media generated by an application may be in response to inputsreceived from the user via movement of the HR headset 105 or the VRinterface device 140. Examples of applications include: gamingapplications, conferencing applications, video playback application, orother suitable applications.

The tracking module 150 calibrates the system environment 100 using oneor more calibration parameters. As further described in conjunction withFIGS. 3-5, the tracking module 150 may adjust one or more calibrationparameters to reduce error in determination of the position of the VRheadset 105. For example, the tracking module 150 adjusts the focus ofthe imaging device 135 to obtain a more accurate position for observedlocators on the VR headset 105. Moreover, calibration performed by thetracking module 150 also account s for information received from the IMU130. Additionally, as discussed in further detail below in conjunctionwith FIGS. 4 and 5, if that tracking of the VR headset 105 is lost(e.g., the imaging device 135 loses line of sight of at least athreshold number of the locators 120), the tracking module 140re-calibrates some or all of the system environments 100. As usedherein, “loss of tracking” may generally refer to a loss of calibrationof the imaging device 135 or the IMU 130, a loss of relative positionsof one or more rigid bodies in the VR headset 105, a loss of position ofthe VR headset 105 relative to the imaging device 135, or somecombination thereof.

Re-calibration of the system environment 100 is generally transparent tothe user. In some embodiments, the tracking module 150 may prompt theuser to move the VR headset 105 to an orientation where one or moresides of the VR headset 105 are visible to the imaging device 135. Forexample, the tracking module 150 prompts the user to look up, to lookdown, to look left, to look right, or look in another specifieddirection so one or more sides of the VR headset 105 are visible to theimaging device 135. Once a threshold number of locators 120 on the VRheadset 105 are imaged by the imaging device 135, the tracking module150 re-establishes calibration. In some embodiments, the tracking module150 may continually calibrate the system environment 100 or calibratesthe system environment 100 at periodic intervals to maintain accuratetracking of the VR headset 105.

The tracking module 150 may calibrate a system environment 100 includinga VR headset 105 comprising one or more rigid bodies (e.g., see FIGS. 2Aand 2B). Additionally, as further described below in conjunction withFIGS. 3 and 5, the calibration may account for a VR headset 105including two rigid bodies that are non-rigidly coupled (e.g., coupledtogether via an elastic band). The two rigid bodies may be a front rigidbody including the IMU 130 that is positioned in front of the user'seyes, and a rear rigid body that is positioned at the rear of the user'shead. This configuration of the front rigid body and the rear rigid bodyallows a user to turn 360 degrees relative to the imaging device 135.However, because the relationship between the front rigid body and therear rigid body is not necessarily fixed, the system environment 100 maylose calibration of the position of the front rigid body relative to therear rigid body. Moreover as discussed in detail below with regard toFIG. 6, in some embodiments, if tracking is lost between multiple rigidbodies in the VR headset 105, the tracking module 150 may offset theposition of a rigid body until re-calibration may occur. In theseinstances, in some embodiments, the tracking module 150 may determine anoffset value to the intermediate estimated position of the VR headset105 and provide it to the IMU 130 as a calibration parameter.Alternatively, the tracking module 150 may adjust a position vectordescribing the relative position of the front rigid body to the rearrigid body by the offset value. In some embodiments, the tracking module150 determines when to re-calibrate based on a measured differencebetween the movement indicated by the locators 120 on the rear rigidbody and the movement predicted using fast calibration data receivedfrom the IMU 130. The tracking module 150 re-calibrates using slowcalibration data including one or more images that include locators 120on the front rigid body and locators on the rear rigid body.

Additionally, the tracking module 150 tracks movements of the VR headset105 using slow calibration data from the imaging device 13. As furtherdescribed below in conjunction with FIG. 3, the tracking module 150determines positions of a reference point of the VR headset 105 usingobserved locators from the slow calibration data and a model of the VRheadset 105. The tracking module 150 also determines positions of areference point of the VR headset 105 using position information fromthe fast calibration data. Additionally, in some embodiments, thetracking module 150 may use portions of the fast calibration data, theslow calibration data, or some combination thereof, to predict a futurelocation of the headset 105. The tracking module 150 provides theestimated or predicted future position of the VR headset 105 to the VRengine 155.

The VR engine 155 executes applications within the system environmentand receives position information, acceleration information, velocityinformation, predicted future positions, or some combination thereof, ofthe VR headset 105 from the tracking module 150. Based on the receivedinformation, the VR engine 155 determines media to provide to the VRheadset 105 for presentation to the user. For example, if the receivedinformation indicates that the user has looked to the left, the VRengine 155 generates media for the VR headset 105 that mirrors theuser's movement in a virtual environment. Additionally, the VR engine155 performs an action within an application executing on the VR console110 in response to an action request received from the VR inputinterface 140 and provides feedback to the user that the action wasperformed. The provided feedback may be visual or audible feedback viathe VR headset 105 or haptic feedback via the VR input interface 140.

FIG. 2A is a wire diagram of one embodiment of a virtual realityheadset. The VR headset 200 is an embodiment of the VR headset 105 andincludes a front rigid body 205 and a band 210. The front rigid body 205includes the electronic display 115 (not shown), the IMU 130, the one ormore position sensors 125, and the locators 120. In the embodiment shownby FIG. 2A, the position sensors 125 are located within the IMU 130, andneither the position sensors 125 nor the IMU 130 are visible to theuser.

The locators 120 are located in fixed positions on the front rigid body205 relative to one another and relative to a reference point 215. Inthe example of FIG. 2A, the reference point 215 is located at the centerof the IMU 130. Each of the locators 120 emit light that is detectableby the imaging device 135. Locators 120, or portions of locators 120,are located on a front side 220A, a top side 220B, a bottom side 220C, aright side 220D, and a left side 220E of the front rigid body 205 in theexample of FIG. 2A.

FIG. 2B is a wire diagram of an embodiment of a VR headset 225 includinga front rigid body 205 and a rear rigid body 230. The VR headset 225shown in FIG. 2B, is an embodiment of the VR headset 105 where the frontrigid body 205 and the rear rigid body 230 are coupled together via theband 210. The band 210 is non-rigid (e.g., elastic), so the front rigidbody 205 is not rigidly coupled to the rear rigid body 210. Thus, therear rigid body 230 may move in relation to the front rigid body 205,and, specifically, move in relation to the reference point 215. Asfurther discussed below in conjunction with FIGS. 3 and 5, the rearrigid body 230 allows the VR console 110 to maintain tracking of the VRheadset 105, even if the front rigid body 205 is not visible to theimaging device 135. Locators 120 on the rear rigid body 230 are locatedin fixed positions relative to one another and relative to the referencepoint 215 on the front rigid body 205. In the example of FIG. 2B, one ormore locators 120, or portions of locators 120, on the rear rigid body230 are located on a front side 235A, a top side 235B, a bottom side235C, a right side 235D, and a left side 235E of the rear rigid body230.

FIG. 3 is a block diagram of one embodiment of the tracking module 150included in the VR console 110. Some embodiments of the tracking module150 have different modules than those described herein. Similarly, thefunctionality described in conjunction with FIG. 3 may be distributedamong the components in a different manner than described herein. In theexample of FIG. 3, the tracking module 150 includes a tracking database310, an initialization module 320, an estimation module 330, a parameteradjustment module 340, and a monitoring module 350.

The tracking database 310 stores information used by the tracking module150 to track one or more VR headsets 105. For example, the trackingdatabase 310 stores one or more headset models, one or more calibrationparameter values, or any other suitable information to track a VRheadset 105. As reference above with respect to FIG. 1, a headset modeldescribes ideal positions of each of the locators 120 with respect toeach other and the reference point 215. Each locator 120 is associatedwith a corresponding model locator in the headset model; hence, a modellocator corresponding to a locator 120 describes an ideal position ofthe locator 120 according to the headset model. Additionally, theheadset model may include information describing changes in modelpositions of the locators 120 or the reference point 215 as a functionof different calibration parameters. In some embodiments, the headsetmodel may describe model positions of locators 120 on a rear rigid body230 with respect to each other, model positions of a rear referencepoint describing a position of the rear rigid body 230, defaultpositions of the rear reference point relative to a reference point 215on the front rigid body 205, default positions of the model locations oflocators 120 on the rear rigid body 230 relative to the reference point215, or some combination thereof.

Calibration parameters are parameters that may be adjusted to affectcalibration of the VR headset 105. Example calibration parametersinclude imaging parameters, IMU parameters, or some combination thereof.Imaging parameters and IMU parameters may be included in the calibrationparameters. Examples of imaging parameters include: focal length, focus,frame rate, ISO, shutter speed, aperture, camera orientation, sourceactivation (in embodiments where the imaging device 135 uses a source toilluminate reflective locators 120), offset of an imaging sensor withrespect to the center of a lens of the imaging device 135, lensdistortion parameters, sensor temperature, or any other parameter usedby the imaging device 135 to output slow calibration data. IMUparameters are parameters controlling collection of the fast calibrationdata. Examples of IMU parameters include: a sample rate of one or moreof the measurement signals from the position sensors 125, an output rateof the fast calibration data, other suitable parameters used by the IMU130 to generate fast calibration data, commands to power the IMU 130 onor off, commands to update the initial position to the current positionof the reference point, offset information (e.g., offset to positionalinformation), or any other suitable information.

The initialization module 320 initializes the system environment 100using information from the tracking database 310, such as calibrationparameters retrieved from the tracking database 310. In embodimentswhere the system environment 100 was not previously calibrated defaultcalibration parameters are retrieved from the tracking database 310. Ifthe system environment 100 was previously calibrated, adjustedcalibration parameters may be retrieved from the tracking database 310.The initialization module 320 provides the retrieved calibrationparameters to the IMU 130 and/or to the imaging device 130.

The estimation module 330 receives slow calibration data and/or fastcalibration data from the VR headset 105 and/or from the IMU 130. Theslow calibration data is received from the imaging device 135 at a slowdata rate (e.g., 20 Hz). In contrast, the fast calibration data isreceived from the IMU 130 at a data rate (e.g., 200 Hz or more) that issignificantly faster than the data rate at which the slow calibrationdata is received. Thus, the fast calibration data may be used todetermine position information of the VR headset 105 between images ofthe VR headset 105 included in the slow calibration data.

Using a headset model from the tracking database 310 and the slowcalibration data from the imaging device 135, the estimation module 330identifies model locators corresponding to one or more locators on theVR headset 135 identified from images captured by the imaging device135. The estimation module 330 extracts locator information from theimages in the slow calibration data, the locator information describingpositions of observed locators 120 relative to each other in a givenimage. For a given image, the locator information describes relativepositions between the observed locators 120 in the image. For example,if an image shows observed locators A, B, and C, the locator informationincludes data describing the relative distances between A and B, A andC, and B and C. As described above, the headset model includes one ormore model positions for the locators on the VR headset 105. Theestimation model 330 compares the relative positions of the observedlocators 120 to the relative positions of the model locators todetermine correspondences between observed locators 120 on the VRheadset 105 and model locators from the headset model. In embodimentswhere calibration is occurring for a VR headset 225 including multiplerigid bodies, model locators corresponding to observed locators on boththe front rigid body 205 and the rear rigid body 230 are identified fromat least one of the images of slow calibration data.

Additionally, based on the headset model and the information describingmodel locators and observed locators 120, the estimation module 330generates estimated positions for observed locators 120. The estimationmodule 330 determines a projection matrix based on the headset model andthe information describing model locators and observed locators 120. Theprojection matrix is a mathematical construct that translates idealpositions of locators 120, described by the headset model, to positionson an image plane, described by the images of the observed locators 120,of the imaging device 135. Thus, the estimation module 330 estimatespositions of observed locators 120 using the projection matrix andpositions of model locators described in the headset model. One or morecalibration parameters may be applied to the projection matrix soadjustments to one or more of the calibration parameters modify theestimated positions of the observed locators 120.

The estimation module 330 also extracts intermediate positioninformation, intermediate velocity information, intermediateacceleration information, or some combination thereof, from the fastcalibration data. As the fast calibration data is received morefrequently than the slow calibration data, information extracted fromthe fast calibration data allows the estimation module 330 to determineposition information, velocity information, or acceleration informationfor time periods between images from the slow calibration data. Anintermediate estimated position information (e.g., an intermediateestimated position) describes a position of the reference point 215 at atime associated with an image, or a time between times associated withan image and a subsequent image from the slow calibration data.Intermediate velocity information describes a velocity vector associatedwith the reference point 215 at a time between a time associated with animage and a time associated with a subsequent image from the slowcalibration data. Intermediate acceleration information describes anacceleration vector associated with the reference point 215 at a timebetween a time associated with an image and a time associated with asubsequent image from the slow calibration data. In some embodiments,the estimation module 330 is configured to obtain the intermediateestimated position information using the intermediate accelerationinformation or from the intermediate velocity information. Theestimation module 330 provides the intermediate position to theparameter adjustment module 340.

The parameter adjustment module 340 adjusts one or more calibrationparameters to adjust the estimated positions until relative distancesbetween the adjusted estimated positions of the observed locators 120and positions of their corresponding model locators are less than athreshold value. If a relative distance between an estimated position ofan observed locator 120 and a position of its corresponding modellocator equals or exceeds a threshold value (e.g., 1 mm), the parameteradjustment module 340 adjusts one or more calibration parameters (e.g.,imaging parameters) until the relative distance is less than thethreshold value. For example, the parameter adjustment module 340modifies one calibration parameter while keeping other calibrationparameters fixed to determine a value for the calibration parameterbeing modified resulting less than a threshold distance between theestimated position of an observed locator 120 and a position of itscorresponding model locator. The parameter adjustment module 340 maythen fix the calibration parameter to the determined value and repeatthe process of modifying values for individual calibration parameterswhile keeping other calibration parameters at constant values untilrelative distances between adjusted estimated positions of at least athreshold number of observed locators 120 and positions of theircorresponding model locators are less than the threshold value. Usingthe adjusted estimated positions of the observed locators 120, theparameter adjustment module 340 generates calibrated positions of thereference point 215 for one or more frames of the slow calibration data.

In embodiments where the VR headset 105 includes two rigid bodies (e.g.,VR headset 225) the parameter adjustment module 340 determines aposition of the rear rigid body 230 relative to the reference point 215on the front rigid body 205. In some embodiments, the parameteradjustment module 340 identifies a rear reference point on the rearrigid body 230 using the observed locators 120 on the rear rigid body230 and their corresponding model locators. The parameter adjustmentmodule 340 then identifies a position of the rear reference pointrelative to the reference point 215 on the front rigid body 205.Alternatively, the VR console 110 identifies the position of eachobserved locator 120 on the rear rigid body 230 relative to thereference point 215 on the front rigid body 205. In some embodiments,the parameter adjustment module 340 generates the calibrated positionsof the reference point 215 responsive to determining that a thresholdnumber of locators are imaged (observed locators) on one or more sidesof each rigid body 205, 230 or a threshold number of locators are imaged(observed locators) on all sides of each rigid body 205, 230. Forexample, the threshold number of locators imaged on a side of a rigidbody 205, 230 is greater than or equal to zero. If the threshold numberof locators is not imaged, the parameter adjustment module 340 mayprompt the user via the VR headset 105 or via another suitable componentto orient the VR headset 105 in a specific direction relative to theimaging device 135 or to continue moving the VR headset 105 until thethreshold number of locators are imaged.

The parameter adjustment module 340 also determines a predictionfunction predicting positions of the reference point 215 and adjusts oneor more calibration parameters until the intermediate estimatedpositions of the reference point 215 from the fast calibration data arewithin a threshold value of the predicted positions of the referencepoint 215. For example, the prediction function is generated by fittinga curve to the series of calibrated positions. The parameter adjustmentmodule 340 then adjusts one or more calibration parameters until adistance between the intermediate estimated positions of the referencepoint 215 and the predicted positions of the reference point 215 is lessthan a threshold value. For example, the parameter adjustment module 340may increase the sample rate of the IMU 140 until the distance betweenthe intermediate estimated positions of the reference point 215 and thepredicted positions of the reference point 215 is 1 mm or less. In otherembodiments, the parameter adjustment module 340 adjusts one or morecalibration parameters so distances between each intermediate estimatedposition and a calibrated position (e.g., CP₁) of the reference point215 associated with the image is less than a distance value between thecalibrated position (e.g., CP₁) of the reference point 215 associatedwith the image and the calibrated position of the reference point 215associated with the subsequent image (e.g., CP₂).

In some embodiments, the parameter adjustment module 340 updates theinitial position of the IMU 130 to be the next calibrated position ofthe reference point 215. As discussed above in conjunction with FIG. 1and below in conjunction with FIG. 6, the IMU 130 collects fastcalibration data relative to positions of the reference point 215previously determined by the IMU 130. Accordingly, drift error increasesthe longer the IMU 130 collects data without updating the initialposition to a calibrated position. The parameter adjustment module 340compares the intermediate estimated positions with an update thresholdvalue. If one or more of the intermediate estimated positions exceed theupdate threshold value, the parameter adjustment module 340 communicatesan instruction to the IMU 130 to update the initial position as theposition associated with the next calibrated position. Alternatively,after determining a calibrated position, the parameter adjustment module340 instructs the IMU 130 to update the initial position to thedetermined calibrated position. The parameter adjustment module 340stores the values for the adjusted calibration parameters in thetracking database 310 and may also provide the adjusted calibrationparameters to other components in the VR console 110.

The monitoring module 350 monitors the system environment 100 for lossof calibration. In various embodiments, the monitoring module 350monitors the relative distances between adjusted estimated positions ofthe observed locators 120 and positions of their corresponding modellocators. If a relative distance between an adjusted estimated positionof an observed locator and a position of its corresponding model locatoris less than a threshold value (e.g., 1 mm), the monitoring module 350provides the calibrated position of the reference point 215 determinedfrom the positions of the observed locators 120 to the VR engine 155. Incontrast, if the relative distance between an observed locator and itscorresponding model locator is more than the threshold value (e.g., 1mm), the monitoring module 350 determines that calibration is lost andprompts the parameter adjustment module 340 to re-calibrate the systemenvironment 100.

To monitor relative distances determined by the parameter adjustmentmodule 340 between intermediate estimated positions and theircorresponding predicted positions. If a distance between a predictedposition and its corresponding intermediate estimated position is lessthan a threshold value (e.g., 1 mm), the monitoring module 350 providesthe intermediate estimated position to the VR engine 155. In someembodiments, the monitoring module 350 may also provide intermediatevelocity information or intermediate acceleration information extractedfrom the fast calibration data to the VR engine 155. In contrast, if thedistance between the predicted position and its correspondingintermediate estimated position is more than the threshold value, themonitoring module 350 determines that calibration is lost and causes thesystem environment 100 to re-establish calibration.

In some instances, locators 120 on the rear rigid body 230 are onlyvisible to the imaging device 135. When only locators 120 on the rearrigid body 230 are visible to the imaging device 135, in someembodiments, if a difference between estimated position of the rearrigid body 230 (e.g., generated from the observed locators 120 on therear rigid body 230) and a predicted position of the rear rigid body 230(e.g., may be generated using fast calibration data) is greater than athreshold value, the monitoring module 350 determines calibration hasbeen lost and causes the system environment 100 to re-establishcalibration. Additionally, if the difference between estimated positionof the rear rigid body 230 and the predicted position of the rear rigidbody 230 is greater than the threshold value, the VR console 110 adjuststhe predicted position of the rear rigid body 230 by a temporary offsetvalue so the difference between the estimated position of the rear rigidbody 230 and the predicted position of the rear rigid body 230 is lessthan the threshold value. The monitoring module 350 may then use thetemporary offset value (or subsequently generated temporary offsetvalues) to more accurately predict the position of the rear rigid body230 until re-calibration may occur between the front rigid body 205 andthe rear rigid body 230. Alternatively, if a difference betweenestimated positions of the locators 120 on the rear rigid body 230 andpositions of their corresponding model locators, relative to thereference point 215, is greater than a threshold value, the monitoringmodule 350 determines calibration has been lost and causes the systemenvironment 100 to re-establish calibration. In some embodiments, whenthe slow calibration data includes an image including a threshold numberof locators on the front rigid body 205 and a threshold number oflocators on the rear rigid body 230, the tracking module 150 beginsre-calibration. Additionally, in some embodiments, once tracking islost, the monitoring module 350 automatically prompts the user to adjustthe VR headset 105 so locators on both the front rigid body 205 and therear rigid body 230 are visible.

Calibrating Virtual Reality Systems

FIG. 4 is a flowchart of one embodiment of a process for calibrating aVR system, such as the system environment 100 described above inconjunction with FIG. 1. In other embodiments, the process includesdifferent, additional, or fewer steps than those depicted by FIG. 4.Additionally, in some embodiments, the steps described in conjunctionwith FIG. 4 may be performed in different orders.

The VR console 110 initializes 410 the system environment using one ormore calibration parameters. For example, the VR console 110 retrievesone or more calibration parameters associated with the VR headset 105from the tracking database 310. In some embodiments, the VR console 110retrieves adjusted calibration parameter values from the trackingdatabase 310 if the imaging device 135 or the IMU 130 were previouslycalibrated for a particular VR headset 105. If the imaging device 135 orthe IMU 130 were not previously calibrated to the VR headset 105, the VRconsole 110 retrieves default calibration parameters from the trackingdatabase 310. The VR console 110 provides the calibration parameters tothe IMU 130 or to the imaging device 135.

The VR console 110 receives 420 slow calibration data from the imagingdevice 135 and fast calibration data from the IMU 130. The slowcalibration data includes a series of images including one or more ofthe locators 120 on the VR headset 105. A locator 120 included in animage from the slow calibration data is referred to herein as an“observed locator.” The fast calibration data may include one or moreintermediate estimated positions of the reference point 215 (e.g., acenter of the IMU 130). In other embodiments, the fast calibration dataincludes intermediate acceleration information and/or intermediatevelocity information from which the VR console 110 determines one ormore intermediate estimated positions of the reference point 215.

Based at least in part on the slow calibration data and a headset model,the VR console 110 identifies 430 model locators, which are locators inthe headset model. The VR console 110 extracts locator informationdescribing positions of observed locators 120 relative to each other inthe from the slow calibration data and compares the locator informationwith a headset model retrieved from the tracking database 310 toidentify 430 model locators that correspond to the observed locators.The model locators are components of the headset model, so identifying430 a model locator associated with an observed locator allows the VRconsole 110 to subsequently compare a position of the observed locatorwith the ideal position, from the headset model of the model locatorassociated with the observed locator.

Using the headset model, the VR console 110 generates 440 estimatedpositions for one or more of the observed locators 120. The headsetmodel describes ideal positioning between the locators 120 and thereference point 215. In various embodiments, the VR console 110 uses theheadset model and the locator information to determine a projectionmatrix for translating ideal positions in the headset model to positionson an image plane of the imaging device 135. The VR console 110 uses theprojection matrix to estimate positions of the observed locations.Hence, the estimated position of an observed locator 120 identifies anideal position of the observed locator 120 on the image plane of theimages from the slow calibration data.

Based at least in part on relative distances between estimated positionsof one or more observed locators 120 and the positions of the modellocators corresponding to the one or more observed locators 120, the VRconsole 110 adjusts 450 one or more calibration parameters that adjustthe estimated positions of the one or more locators 120 so a relativedistance between estimated positions of observed locators 120 andpositions of their corresponding model locators from the headset modelare less than a threshold value (e.g., 1 mm). Adjusting calibrationparameters affects the projection matrix (e.g., changing focal length,etc.), so changing one or more calibration parameters may affect theestimated positions of the observed locators 120. If the distancesbetween the estimated positions of the observed locators 120 and thepositions of their corresponding model locators equals or exceeds thethreshold value, in one embodiment, the VR console 110 adjusts 450 onecalibration parameter while keeping other calibration parameters fixedto determine a value for the calibration parameter being adjusted thatresults in a distance between the estimated position of an observedlocator 120 and the position of its corresponding model locator beingless than the threshold value. The calibration parameter may then befixed to the determined value, while another calibration parameter ismodified so the distance between an estimated position of an additionallocator 120 and an additional position of a model locator correspondingto the additional locator is less than the threshold value. Variouscalibration parameters may be adjusted 450 as described above sorelative distances between adjusted estimated positions of at least athreshold number of observed locators 120 and positions of theircorresponding model locators are less than the threshold value. If thedistances between estimated positions of at least a threshold number ofthe observed locators 120 and positions of their corresponding modellocators are less than the threshold value, the calibration parametersare not adjusted 450.

The VR console 110 determines 460 whether a threshold number of theobserved locators 120 are from each side of the front rigid body 205(i.e., the front side 220A, the top side 220B, the bottom side 220C, theright side 220C, and the left side 220D). In embodiments where the VRheadset 105 includes multiple rigid bodies, the VR console 110 generatesthe calibrated positions of the reference point 215 responsive todetermining that a threshold number of locators are imaged (observedlocators) on one or more sides of each rigid body 205, 230 or responsiveto determining that a threshold number of locators are imaged (observedlocators) on all sides of each rigid body 205, 230. If the thresholdnumber of observed locators 120 are associated with each side, the VRconsole 110 generates 470 calibrated positions of the reference point215 for one or more frames of the slow calibration data using theadjusted estimated positions of the observed locators. If the thresholdnumber of observed locators 120 are not associated with each side, theVR console 110 may communicate a prompt to the user via the VR headset105 or another component to reposition the VR headset 105 so that slowcalibration data including locators from one or more sides of the VRheadset 150 may be captured.

The VR console 110 further adjusts 480 one or more calibrationparameters until intermediate estimated positions of the VR headset 105received from the fast calibration data are within a threshold distanceof predicted positions for the VR headset 105 or the reference point215, where the predicted positions are determined from the calibratedpositions of the reference point 215 associated with various images fromthe slow calibration data. In some embodiments, the VR console 110determines a predicted position of the reference point 215 by generating(e.g., via curve fitting) a prediction function using calibratedpositions of the reference point 215 associated with different imagesfrom the slow calibration data. The VR console 110 adjusts one or moreof the calibration parameters until the distances between theintermediate estimated positions of the reference point 215 and thepredicted positions of the reference point 215 are less than a thresholddistance. For example, the VR console 110 may increase the sample rateof the IMU 130 until the distances between the intermediate estimatedpositions of the reference point 215 and the predicted positions of thereference point 215 are all 1 mm or less or until distances between atleast a threshold number of intermediate estimated positions of thereference point 215 and predicted positions of the reference point 215are less than 1 mm. In other embodiments, the VR console 110 determinesa predicted position of the reference point 215 as a position between acalibrated position of the reference point 215 associated with an imagefrom the slow calibration data and a calibrated position of thereference point 215 associated with a subsequent image from the slowcalibration data. The VR console 110 then adjusts 480 one or morecalibration parameters so distances between each intermediate estimatedposition and a calibrated position (e.g., CP₁) of the reference point215 associated with the image is less than a distance between thecalibrated position (e.g., CP₁) of the reference point 215 associatedwith the image and the calibrated position of the reference point 215associated with the subsequent image (e.g., CP₂). Additionally, the VRconsole 110 may update the initial position of the IMU 130 to be thecalibrated position of the reference point 215.

In some embodiments, the VR console 110 stores the values for theadjusted calibration parameters in the tracking database 310 or providesthe values for the adjusted calibration parameters to other componentsin the VR console 110. The adjusted calibration values may reducecalibration times for subsequent operations of the system environment100, improving user experience.

The VR console 110 monitors 490 the system environment 100 for loss ofcalibration. For example, the VR console 110 monitors the relativedistances between adjusted estimated positions of the observed locators120 and positions of their corresponding model locators. If a relativedistance between an adjusted estimated position of an observed locator120 and a position of its corresponding model locator is less than athreshold value (e.g., 1 mm), the VR console 110 provides the calibratedposition to the VR engine 155. In contrast, if the relative distancebetween an estimated position of an observed locator and a position ofits corresponding model locator is greater than (or equals or exceeds)than the threshold value (e.g., 1 mm), the VR console 110 determinesthat calibration is lost, receives 420 slow calibration data and fastcalibration data and performs the above-identified functions tore-calibrate the system environment 110.

Additionally, the VR console 110 monitors 490 the relative distancesbetween intermediate estimated positions of the reference point 215 andpredicted positions of the reference point 215. For example, if adistance between a curve of predicted positions of the reference point215 and an intermediate estimated position of the reference point 215 isless than a threshold distance (e.g., 1 mm), the VR console 110 providesthe intermediate estimated position to the VR engine 155. In someembodiments, the VR console 110 may also provide intermediate velocityinformation or intermediate acceleration information extracted from thefast calibration data to the VR engine 155. In contrast, if the distancebetween the predicted position of the reference point 215 and anintermediate estimated position of the reference point 215 is greaterthan or equals or exceeds the threshold distance, the VR console 110determines that calibration is lost, receives 420 slow calibration dataand fast calibration data and performs the above-identified functions tore-calibrate the system environment 100.

In some embodiments, the IMU 130 and the imagining device 135 may becalibrated simultaneously. To simultaneously calibrate the IMU 130 andthe imaging device 135, the VR console 110 estimates positions of thereference point 215 for a series of images using estimated positions ofthe observed locators. Additionally, the VR console 110 uses fastcalibration data including the intermediate estimated positions of thereference point 215 at particular time values corresponding to images inthe slow calibration data when calibrating the IMU 130 and the imagingdevice 135. When simultaneously adjusting calibration parameters of theIMU 130 and of the imaging device 135, the VR console 110: (1) adjustsestimated positions of observed locators so a relative distance betweenthe adjusted estimated positions of the observed locators and positionsof their corresponding model locaters are less than a threshold value;and (2) adjusts the estimated positions for the reference point so arelative distance between the estimated positions for the referencepoint at the particular time values corresponding to images in the slowcalibration data and the positions of a model reference point determinedfrom the model locators is less than the threshold value.

FIG. 5 is a flowchart illustrating one embodiment of a process forre-establishing calibration between two rigid bodies of a virtualreality headset 225 included in the system environment 100. In otherembodiments, the process includes different, additional, or fewer stepsthan those depicted by FIG. 5. Additionally, in some embodiments, thesteps described in conjunction with FIG. 5 may be performed in differentorders.

The VR console 110 receives 510 slow calibration data including imagesshowing a front threshold number of locators 120 on a front rigid body205 and a rear threshold number (e.g., at least one) of locators 120 ona rear rigid body 230 of the VR headset 225. A locator 120 included inan image from the slow calibration data is referred to herein as an“observed locator.” As described above in conjunction with FIGS. 2-4,the VR console 110 receives 150 the slow calibration data from theimaging device 135 and the fast calibration data from the IMU 130. Thefast calibration data may also include intermediate accelerationinformation and/or intermediate velocity information from which the VRconsole 110 determines one or more intermediate estimated positions ofthe reference point 215 of the VR headset 225.

Based at least in part on the slow calibration data and a headset model,the VR console 110 identifies 520 model locators, which are locators inthe headset model. The VR console 110 extracts locator informationdescribing positions of observed locators 120 relative to each otherfrom the slow calibration data and compares the locator information witha headset model retrieved from the tracking database 310 to identify 520model locators that correspond to the observed locators 120. In at leastone of the images model locators are identified that correspond toobserved locators on both the front rigid body 205 and the rear rigidbody 230 of the VR headset 225. The model locators are components of theheadset model, so identifying 520 a model locator associated with anobserved locator allows the VR console 110 to subsequently compare aposition of the observed locator with the ideal position, from theheadset model of the model locator associated with the observed locator.

Using the headset model, the VR console 110 generates 530 estimatedpositions for one or more of the observed locators 120. The headsetmodel describes ideal positioning between the locators 120 and thereference point 215. In various embodiments, the VR console 110 uses theheadset model and the locator information to determine a projectionmatrix for translating ideal positions in the headset model to positionson an image plane of the imaging device 135. The VR console 110 uses theprojection matrix to estimate positions of the observed locators 120.Hence, the estimated position of an observed locator 120 identifies anideal position of the observed locator 120 on the image plane of theimages from the slow calibration data.

Based at least in part on relative distances between estimated positionsof one or more observed locators 120 and the positions of the modellocators corresponding to the one or more observed locators 120, the VRconsole 110 adjusts 540 relative distance between estimated positions ofobserved locators on the first rigid body 205 and positions of theircorresponding model locators are less than a threshold value (e.g., 1mm). Adjusting calibration parameters affects the projection matrix(e.g., changing focal length, etc.), so changing one or more calibrationparameters may affect the estimated positions of the observed locators120. If the distances between the estimated positions of the observedlocators 120 and the positions of their corresponding model locatorsequals or exceeds the threshold value, in one embodiment, the VR console110 adjusts 540 one calibration parameter while keeping othercalibration parameters fixed to determine a value for the calibrationparameter being adjusted that results in a distance between theestimated position of an observed locator 120 and the position of itscorresponding model locator being less than the threshold value.Adjustment 540 of calibration parameters is further described above inconjunction with FIG. 4. If the distances between estimated positions ofat least a threshold number of the observed locators 120 and positionsof their corresponding model locators are less than the threshold value,the calibration parameters are not adjusted 540.

After adjusting 540 calibration parameters so at least a thresholdnumber of relative distances between the estimated positions of theobserved locators and the positions of their corresponding modellocators are less than the threshold value, the VR console 110 generates550 calibrated positions of the reference point 215 associated with oneor more images of the slow calibration data using the adjusted estimatedpositions of the observed locators 120. In some embodiments, the VRconsole 110 generates the calibrated positions of the reference point215 responsive to determining that a threshold number of locators areimaged (observed locators) on one or more sides of each rigid body 205,230 or determining that a threshold number of locators are imaged(observed locators) on all sides of each rigid body 205, 230. If thethreshold number of locators (on a side of a rigid body 205, 230 or onall sides of each rigid body 205, 230) is not imaged, the VR console 110may prompt the user via the VR headset 105 or via another suitablecomponent to orient the VR headset 105 in a specific direction relativeto the imaging device 135 or to continue moving the VR headset 105 untilthe threshold number of locators are imaged.

The VR console 110 also determines 560 a position of the rear rigid body230 relative to the reference point 215. In some embodiments, the VRconsole 110 identifies a rear reference point on the rear rigid body 230using the observed locators 120 and their corresponding model locators.The VR console 110 then identifies the position of the rear referencepoint relative to the reference point 215 on the front rigid body 205such that the rear reference point is positioned relative to thereference point 215 by a position vector. Alternatively, the VR console110 identifies the position of each observed locator on the rear rigidbody 230 relative to the reference point 215, so positions of eachobserved locator on the rear rigid body 230 are positioned relative tothe reference point 215 by their own position vector.

The VR console 110 adjusts 570 one or more calibration parameters so theintermediate estimated positions of the reference point 215 are within athreshold distance of predicted positions of the reference point 215.Adjustment of calibration parameters so intermediate estimated positionsof the reference point 215 are within a threshold value of predictedpositions of the reference point is further described above inconjunction with FIG. 4. After adjusting 570 one or more calibrationparameters, the VR console 110 monitors 580 for loss of calibration ofthe system environment 100, as described above in conjunction with FIG.4.

When monitoring 580 for loss of calibration, the VR console 110 usesimages from the slow calibration data that may include observedpositions of locators 120 on the first rigid body 205, on the rear rigidbody 230, or on some combination thereof. In some embodiments, thethreshold value between a position of an observed locator 120 and aposition of its corresponding model locator may differ based on therigid body on which the observed locator 120 is located. For example,the threshold value may be 1 mm for observed locators 120 on the frontrigid body 205 and 2 mm for observed locators 120 on the rear rigid body230.

Additionally, in some scenarios, the imaging device 135 is unable toview locators 120 on the front rigid body 205, but is able to viewlocators on the rear rigid body 230. In these scenarios, tracking ismonitored using the process described below with respect to FIG. 6.

When the slow calibration data includes an image including a thresholdnumber of locators on the front rigid body 205 and a threshold number oflocators on the rear rigid body 230, the steps described above inconjunction with FIG. 5 are repeated to re-establish calibration of thesystem environment 100. In some embodiments, when tracking is lost, theVR console 110 automatically prompts the user to adjust the VR headset105 so locators on both the front rigid body 205 and on the rear rigidbody 230 are visible to the imaging device 135. The prompt presented tothe user may provide the user with specific instructions to position theVR headset 105 so locators on both the front rigid body 205 and on therear rigid body 230 are visible to the imaging device 135.

FIG. 6 is a flowchart illustrating one embodiment of a process formaintaining a positional relationship between two rigid bodies of avirtual reality headset 225 included in the system environment 100. Inother embodiments, the process includes different, additional, or fewersteps than those depicted by FIG. 6. Additionally, in some embodiments,the steps described in conjunction with FIG. 6 may be performed indifferent orders.

The VR console 110 receives 610 slow calibration data from the imagingdevice 135. The slow calibration data includes a series of images thatincludes an image associated with an image time value and having onlyobserved locators 120 on the rear rigid body 230 visible to the imagingdevice 135. An image time value is a time value when the image wascaptured by the imaging device 135. Additionally, the VR console 110receives 620, from the IMU 130, fast calibration data that includesintermediate estimated positions of a reference point 215 for a seriesof time values that includes the image time value.

Based on the slow calibration data, the VR console 110 determines 630 anobserved position of the rear rigid body 230 at the image time value. Todetermine 620 the observed position of the rear rigid body 230, the VRconsole 110 extracts locator information describing positions ofobserved locators 120 on the rear rigid body 230 relative to each otherfrom the slow calibration data and compares the locator information witha headset model retrieved from the tracking database 310 to identifymodel locators corresponding to the observed locators 120. Afteridentifying model locators, the VR console 110 determines the observedlocators 120 corresponding to each model locator and determines a rearreference point for the rear rigid body 230 using the positions of theobserved locators 120. In some embodiments, the observed position of therear rigid body 230 is the position of the rear reference point. Inalternate embodiments, the observed position of the rear rigid body 230may be observed positions of one or more of the observed locators.

The VR console 110 determines 640 a predicted position of the rear rigidbody 230 at the image time value using the fast calibration data and aposition vector. The position vector describes a calibrated offsetbetween the front rigid body 205 and the rear rigid body 230. Forexample, the position vector describes a calibrated offset between thereference point 215 associated with the front rigid body 205 and a rearreference point associated with the rear rigid body 230. Additionally,in some embodiments, the position vector may include one or moresub-vectors that each describe relative calibrated offsets between thereference point 215 and different locators on the rear rigid body 230.

From the fast calibration data, the VR console 110 determines anintermediate estimated position of the reference point 215 on the frontrigid body 205. In some embodiments, the VR console 110 determines thepredicted position of the rear rigid body 230 as a position relative tothe position of the reference point 215 based on the position vector.For example, the position vector identifies a relative positioning of arear reference point on the rear rigid body 230 to the reference point215. Alternatively, the position vector identifies the relativepositioning of one or more locators 120 (including the observedlocators) on the rear rigid body 230 relative to the reference point215.

The VR console 110 determines 650 whether a difference between theobserved position and the predicted position is greater than a thresholdvalue (e.g., 1 mm). If the difference is less than the threshold value,tracking of the VR headset 225 is maintained and slow calibration datais received 610, and the process proceeds as described above. However,if the difference between the observed position of the rear rigid body230 exceeds the threshold value, the VR console 110 determines trackingof the VR headset 105 is lost and adjusts 660 the predicted position byan offset value. The offset value is determined so the differencebetween the between the observed position of the rear rigid body 230 andthe predicted position of the rear rigid body 230 is less than thethreshold value. For example, the VR console 110 uses the positionvector modified by the offset value to more accurately determine theposition of the rear rigid body 230 from the fast calibration data.Alternatively, the VR console 110 communicates an instruction to the IMU130 to offset the estimated intermediate positions based on the offsetvalue without modifying the position vector.

Based on the fast calibration data and the adjusted vector, the VRconsole 110 determines 670 subsequent predicted positions of the rearrigid body until re-calibration occurs (e.g., as further described abovewith respect to FIG. 5). In some embodiments, when tracking is lost, theVR console 110 prompts the user to adjust the VR headset 105 so locatorson both the front rigid body 205 and on the rear rigid body 230 arevisible to the imaging device 135. The prompt presented to the user mayprovide the user with specific instructions to position the VR headset105 so locators on both the front rigid body 205 and on the rear rigidbody 230 are visible to the imaging device 135 to facilitatere-calibration described in detail above with reference to FIG. 5.

FIG. 7 illustrates an example graph 700 illustrating a series ofcalibrated positions of a virtual reality headset 105. In FIG. 7, thevertical axis represents position, and the horizontal axis representstime. The graph 700 includes a series of calibrated positions 710A-C ofa reference point of the VR headset 105 at times, T₁, T₂, and T₃,respectively. The graph 700 also includes a series of intermediateestimated positions 715A-D and 720A-H of the reference point. Thecalibrated positions 710A-C are generated using slow calibration datafrom an imaging device 135 and the intermediate estimated positions715A-D and 720 A-H are generated using fast calibration data from an IMU130 included on a VR headset 105. Note, the relative times scales of thecalibrated positions 710A-C and of the intermediate estimated positions715A-D are different, and that intermediate estimated positions 715A-Dand 720A-H are determined more frequently than the calibrated positions710A-C.

The graph 700 shows a predicted position curve 725 described by aprediction function describing the predicted position of the referencepoint. The prediction function is generated by fitting a curve to thecalibrated positions 710A-C and determining a function that describesthe fitted curve. Any suitable method may be used to determine theposition function from the calibrated positions 710A-C.

In the example of FIG. 7, the intermediate estimated positions 715A-Dare the initial intermediate estimated positions determined using thefast calibration data prior to adjustment of the calibration parameters.Intermediate estimated position 715A is relatively close to thepredicted position curve in FIG. 7, but as time progresses, theintermediate estimated positions move farther away from the predictedposition curve 725, with the intermediate estimated position 715D inFIG. 7 being the farthest from the predicted position curve 725. Thedifference between the predicted position curve and the intermediateestimated position may be attributed to a combination of actual usermovements, drift error, as well as additional factors. As discussedabove, because the IMU 130 determines an intermediate estimated positionrelative to a previously determined position, the error compounds,resulting in larger deviation between the predicted position curve 725and intermediate estimated positions 15 over time. To account for drifterror, the VR console 110 may update an initial position of the IMU 130as the subsequent calibration position. The IMU 130 then generates fastcalibration with respect to the updated initial position andintermediate estimated positions determined after the initial position.In this embodiment, the VR console 110 updates the initial point as thecalibrated position 710B.

Another way to reduce error associated with the intermediate estimatedpositions is by increasing the frequency that the intermediate estimatedpositions are determined. In the example of FIG. 7, the VR console 110determined the intermediate estimated positions 720 A-H at twice thefrequency of the intermediate estimated positions 715 A-D, resulting ina smaller difference between the intermediate estimated positions 720A-H and the predicted position curve 725.

Summary

The foregoing description of the embodiments of the disclosure has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

The foregoing description of the embodiments of the disclosure has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the disclosure be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

What is claimed is:
 1. A system comprising: a virtual reality (VR)headset including a front rigid body, a rear rigid body, a plurality oflocators located on each of the front rigid body and the rear rigidbody, and an inertial measurement unit (IMU) configured to output fastcalibration data comprising one or more intermediate estimated positionsof a reference point on the VR headset, the estimated positionsdescribing an estimated position of the VR headset wherein each of theintermediate estimated positions is separated from a subsequentintermediate estimated position by a position time value wherein thefront rigid body is non-rigidly coupled to the rear rigid body via anelastic band; an imaging device external to the VR headset configured tocapture images of the VR headset and output slow calibration dataincluding a series of images showing portions of observed locators ofthe plurality of locators located on the VR headset, each imageseparated from a subsequent image in the series by an image time valuethat is larger than the position time value; and a VR consolecomprising: a processor, and a memory coupled to the processor andincluding instructions that, when executed by the processor, cause theprocessor to track the VR headset using the slow calibration data andthe fast calibration data.
 2. The system of claim 1, wherein the IMUincludes a three-axis gyroscope to measure angular velocity.
 3. Thesystem of claim 1, wherein the IMU includes a three-axis accelerometer.4. The system of claim 1, wherein the IMU includes a three-axismagnetometer.
 5. The system of claim 1, wherein the plurality oflocators are arranged in a pattern on the VR headset that isnon-coplanar.
 6. The system of claim 1, wherein the plurality oflocators are light emitting diodes (LEDs).
 7. The system of claim 6,wherein the LEDs are modulated to maintain one out of two or morepredetermined brightness levels during a time interval.
 8. The system ofclaim 7, wherein the modulation of the LEDs is selected from a groupconsisting of: amplitude modulation, frequency modulation, and anycombination thereof.
 9. The system of claim 7, wherein the plurality oflocators emit in the infrared band and an outer surface of the VRheadset is transparent in the infrared band but opaque in the visibleband.
 10. The system of claim 6, wherein the LEDs emit light within aspecific band selected from a group consisting of a visible band and aninfrared band.
 11. The system of claim 1, further comprising calibrationinstructions that, when executed by the processor, cause the processorto: identify model locators each corresponding to a locator on the VRheadset and included in at least one image from the slow calibrationdata using a stored headset model associated with the VR headset;generate estimated positions of one or more of the locators on the VRheadset and included in at least one image from the slow calibrationdata using the headset model; adjust one or more calibration parametersto adjust the estimated positions so a relative distance between theadjusted estimated positions of one or more of the locators on the VRheadset and included in at least one image from the slow calibrationdata and positions of their corresponding model locators are less than athreshold value; generate calibrated positions of the reference pointbased at least in part on the adjusted estimated positions of one ormore of the locators on the VR headset and included in at least oneimage from the slow calibration data, a calibrated position associatedwith an image from the slow calibration data; determine one or morepredicted positions of the reference point based at least in part on thecalibrated positions of the reference point, a predicted positionassociated with a time between subsequent images from the slowcalibration data; and adjust one or more of the calibration parametersso the intermediate estimated positions of the reference point arewithin a threshold distance of the determined predicted positions of thereference point.
 12. The system of claim 1, further comprisingcalibration instructions that, when executed by the processor, cause theprocessor to: determine an observed position of the rear rigid body forthe particular image time value using the slow calibration data;determine a predicted position of the rear rigid body for the particularimage time value using the fast calibration data and a position vectordescribing a calibrated offset between the front rigid body and the rearrigid body; determine that a difference between the observed positionand the predicted position is greater than a threshold value; responsiveto a determination that a difference between the observed position ofthe rear rigid body and the predicted position of the rear rigid body isgreater than a threshold value, adjust the position vector by an offsetvalue so the difference between the observed position and the predictedposition is less than the threshold value; and determine a subsequentpredicted position of the rear rigid body for an image from the seriesof images associated with a subsequent image time value occurring afterthe image time value based on the fast calibration data and the adjustedposition vector.
 13. A system comprising: a virtual reality (VR) headsetincluding a front rigid body, a rear rigid body, a plurality of locatorslocated on each of the front rigid body and the rear rigid body, and aninertial measurement unit (IMU) configured to output fast calibrationdata comprising one or more intermediate estimated positions of areference point on the VR headset, the estimated positions describing anestimated position of the VR headset wherein each of the intermediateestimated positions is separated from a subsequent intermediateestimated position by a position time value wherein the front rigid bodyis non-rigidly coupled to the rear rigid body via an elastic band; animaging device configured to output slow calibration data including aseries of images showing portions of observed locators of the pluralityof locators locator, on the VR headset, each image separated from asubsequent image in the series by an image time value that is largerthan the position time value; and a VR console comprising: a processor,and a memory coupled to the processor and including instructions that,when executed by the processor, cause the processor to track the VRheadset using the slow calibration data and the fast calibration data.14. The system of claim 13, wherein the plurality of locators includes alocator that is an LED that is modulated to maintain one out of two ormore predetermined brightness levels during a time interval.
 15. Thesystem of claim 14, wherein the modulation of the LED is selected from agroup consisting of: amplitude modulation, frequency modulation, and acombination thereof.
 16. The system of claim 14, wherein the LED emitslight within a specific band selected from a group consisting of avisible band and an infrared band.
 17. The system of claim 16, whereinthe LED emits in the infrared band and an outer surface of the VRheadset is transparent in the infrared band but opaque in the visibleband.
 18. The system of claim 13, wherein the VR headset includes afront rigid body and a rear rigid body each having one or more locators,and the front rigid body includes the IMU.